Papers with sentence prediction
A Multi-Task Learning Framework for Extracting Bacteria Biotope Information (D19-57)
Copied to clipboard
| Challenge: | Existing methods to extract information from unstructured text are slow or expensive to get. |
| Approach: | They propose a multi-task transfer multi-learning method for Bacteria Biotope rel+ner task . they use BERT and pre-train it using mask language models and next sentence prediction . |
| Outcome: | The proposed method achieves the best performance on all metrics including slot error rate, precision and recall in the Bacteria Biotope rel+ner subtask. |
NCPrompt: NSP-Based Prompt Learning and Contrastive Learning for Implicit Discourse Relation Recognition (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent prompt learning methods have demonstrated success in IDRR, but they fail to fully exploit critical semantic features shared among various forms of templates. |
| Approach: | They propose an NSP-based prompt learning and contrastive learning method for IDRR that transforms the IDRR task into a next sentence prediction task. |
| Outcome: | The proposed model can be used to classify the discourse relation sense between argument pairs without an explicit connective. |
Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is widely used in natural language processing for self-supervised learning of text representations. |
| Approach: | They propose to use token-level classification tasks as main pretraining objectives instead of Masked language modeling (MLM) . Empirical results show that pretraining a model with 41% of the BERT-BASE’s parameters, BERT MEDIUM results in only a 1% drop in GLUE scores with their best objective. |
| Outcome: | Empirical results show that the proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture. |
On Losses for Modern Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Devlin et al. ( 2018) released a transformer network (BERT) pre-training over two tasks: masked language modelling (MLM) and next sentence prediction (NSP). |
| Approach: | They clarify NSP's effect on BERT pre-training and explore ways to include multiple tasks into pre-train. |
| Outcome: | The proposed framework outperforms BERTBase on the GLUE benchmark using fewer than a quarter of training tokens. |
Muppet: Massive Multi-task Representations with Pre-Finetuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows gains from pre-training and fine-tuning that are multi-task . but it can be difficult to know which intermediate tasks will best transfer . |
| Approach: | They propose a large-scale learning stage for pre-finetuning between pre-training and fine-tun. |
| Outcome: | The proposed model improves performance on pretrained discriminators and generation models on a wide range of tasks while improving sample efficiency during fine-tuning. |
SCDE: Sentence Cloze Dataset with High Quality Distractors From Examinations (2020.acl-main)
Copied to clipboard
| Challenge: | Experimental results show that there is a significant performance gap between advanced models (72%) and humans (87%) Cloze datasets are convenient either to be generated automatically or by annotators. |
| Approach: | They propose to use a dataset to evaluate the performance of computational models through sentence prediction. |
| Outcome: | The proposed model fills up multiple blanks in a passage from a shared candidate set with distractors designed by English teachers. |